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ON THE ENUMERATION OF TANGLEGRAMS AND TANGLED GRAINS 


SARA C. BILLEY, MATJAZ KONVALINKA, AND FREDERICK A. MATSEN IV 


Abstract. Tanglegrams are a special class of graphs appearing in applications concerning cospeciation and 
coevolution in biology and computer science. They are formed by identifying the leaves of two rooted binary 
trees. We give an explicit formula to count the number of distinct binary rooted tanglegrams with n matched 
vertices, along with a simple asymptotic formula and an algorithm for choosing a tanglegram uniformly at 
random. The enumeration formula is then extended to count the number of tangled chains of binary trees 
of any length. This includes a new formula for the number of binary trees with n leaves. We also give a 
conjecture for the expected number of cherries in a large randomly chosen binary tree and an extension of 
this conjecture to other types of trees. 


1. Introduction 

Tanglegrams are graphs obtained by taking two binary rooted trees with the same number of leaves and 
matching each leaf from the tree on the left with a unique leaf from the tree on the right. This construction is 
used in the study of cospeciation and coevolution in biology. For example, the tree on the left may represent 
the phylogeny of a host, such as gopher, while the tree on the right may represent a parasite, such as louse m, 
m page 71]. One important problem is to reconstruct the historical associations between the phytogenies 
of host and parasite under a model of parasites switching hosts, which is an instance of the more general 
problem of cophylogeny estimation. See [Biiiiin] for applications in biology. Diaconis and Holmes have 
previously demonstrated how one can encode a phylogenetic tree as a series of binary matchings , which 
is a distinct use of matchings from that discussed here. 

In computer science, the Tanglegram Layout Problem (TL) is to find a drawing of a tanglegram in the 
plane with the left and right trees both given as planar embeddings with the smallest number of crossings 
among (straight) edges matching the leaves of the left tree and the right tree [2]. These authors point out 
that tanglegrams occur in the analysis of software projects and clustering problems. 

In this paper, we give the exact enumeration of tanglegrams with n matched pairs of vertices, along with 
a simple asymptotic formula and an algorithm for choosing a tanglegram uniformly at random. We refer to 
the number of matched vertices in a tanglegram as its size. Furthermore, two tanglegrams are considered to 
be equivalent if one is obtained from the other by replacing the tree on the left or the tree on the right by 
isomorphic trees. For example, in Figure [1] the two non-equivalent tanglegrams of size 3 are shown. 



Figure 1. The tanglegrams of size 3. 

We state our main results here postponing some definitions until Section [2l The following is our main 
theorem. 
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Theorem 1. The number of tanglegrams of size n is 

^ Ili=2 H-1" '^I!(A)) - l) 

tn — / ^ 5 

A 

where the sum is over binary partitions of n and z\ is defined by Equation O- 


The first 10 terms of the sequence starting at n = 1 are 

1,1,2,13,114,1509,25595,535753,13305590,382728552, 
see [T71 A258620] for more terms. 


Example. The binary partitions of n = 4 are (4), (2, 2), (2,1,1) and (1,1,1,1), so 

32 ,-^2 52.32 . i2 


1 32 

" 4 + ¥ 


4 24 

as shown in Figure O It takes a computer only a moment to compute 


= 13 


t42 = 33889136420378480492869677415186948305278176263020722832251621520063757 

and under a minute to compute all 3160 integer digits of tiooo using a recurrence based on Theorem [T] given 
in Section [51 






Figure 2. The 13 tanglegrams of size 4. 


We use the main theorem to study the asymptotics of the sequence . It turns out that 

tn eU"-! 

n! Trn^ 

see Corollary |S] for an explanation and better estimates. 

A side result of the proof is a new formula for the number of inequivalent binary trees, called the 
Wedderburn-Etherington numbers [13 A001190]. 

Theorem 2. The number of inequivalent binary trees with n leaves is 

^ nii^2^ -^^(A)) - 1) 

” ~ A 

where the sum is over binary partitions of n. 

A tangled chain is an ordered sequence of k binary trees with matchings between neighboring trees in 
the sequence. For k = 1, these are inequivalent binary trees, and for k = 2, these are tanglegrams, so the 
following generalizes Theorems [3 and [3 

In terms of computational biology, tangled chains of length k formalize the essential input to a variety of 
problems on k leaf-labeled (phylogenetic) trees (e.g. [M])- 
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Figure 3. The tangled chains of length 3 for n = 3. 


Theorem 3. The number of ordered tangled chains of length k for n is 

(2(Ai H-+ A^(a)) - l) 

where the sum is over binary partitions of n. 

Example. For n = k = 3, we have partitions (2,1) and (1,1,1), and the theorem gives 


as shown in Figure|3l For fc = 3, the number of tangled chains on trees with n leaves gives rise to the sequence 
starting 

1,1,5,151,9944,1196991,226435150,61992679960,23198439767669,11380100883484302. 

See [ITl A258486] for more terms. 


From the enumerative point of view, it is also quite natural to ask how likely a particular tree T is to 
appear on one side or the other of a uniformly selected tanglegram. In Section [71 we give a simple explicit 
conjecture for the asymptotic growth of the expected number of copies of T on one side of a tanglegram as 
a function of T and the size of the tanglegram. For example, the cherries of a binary tree are pairs of leaves 
connected by a common parent. We conjecture that the expected number of cherries in one of the binary 
trees of a tanglegram of size n chosen in the uniform distribution is n/4. 

Further discussion of the applications of tanglegrams along with several variations on the theme are 
described in m- In particular, tanglegrams can be used to compute the subtree-prune-regraft distance 
between two binary trees. 

The paper proceeds as follows. In Section |2l we define our terminology and state the main theorems. We 
prove the main theorems in Section 0 Section [4] contains an algorithm to choose a tanglegram uniformly at 
random for a given n. In SectionjSl we give several asymptotic approximations to the number of tanglegrams 
with increasing accuracy and complexity. In Section [51 we give a recursive formula for both the number of 
tanglegrams and for tangled chains. We conclude with several open problems and conjectures in Section [71 


2. Background 

In this section, we recall some vocabulary and notation on partitions and trees. This terminology can 
also be found in standard textbooks on combinatorics such as [72] . We use these terms to give the formal 
definition of tanglegrams and the notation used in the main theorems. 

A partition A = (Ai, A2 ,..., A^) is a weakly decreasing sequence of positive integers. The length ^(A) of a 
partition is the number of entries in the sequence, and |A| denotes the sum of the entries of A. We say A is a 
binary partition if all its parts are equal to a nonnegative power of 2. Binary partitions have appeared in a 
variety of contexts, see for instance in [HUIlllI] and [m A000123]. When writing partitions, we sometimes 
omit parentheses and commas. 

If A is a nonempty binary partition with nii occurrences of the letter 2* for each i, we also denote A by 
(l'"o,2'"i,4'"^8'"3,...,(2^)'"0 where 2^ = Ai is the maximum value in A. Given A = (I'^o, 2'"i,..., (2^)™!), 
let z\ denote the product 

(1) ZA = l'"°2™i •••(2^)™^TOo!mi!m2!---TOj!. 

The numbers zx are well known since the number of permutations in ©„ with cycle type A is nl/zx [TT] Prop. 
1.3.2]. For example, for A = 44211 = (l^, 2^,42), za = 1^ • 2G 42 • 2! • 1! • 2! = 128. 
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A rooted tree has one distinguished vertex assumed to be a common ancestor of all other vertices. The 
neighbors of the root are its children. Each vertex other than the root has a unique parent going along the 
path back to the root, the other neighbors are its children. In a binary tree, each vertex either has two 
children or no children. A vertex with no children is a leaf, and a vertex with two children is an internal 
vertex. Two binary rooted trees with labeled leaves are said to be equivalent if there is an isomorphism from 
one to the other as graphs mapping the root of one to the root of the other. Let be the set of inequivalent 
binary rooted trees with n > 1 leaves, and let be the number of elements in the set Bn. The sequence of 
6 „’s for n > 1 begins 

1,1,1,2,3,6,11,23,46,98. 

We can inductively define a linear order on rooted trees as follows. We say that T > 5 if either: 

• T has more leaves than S 

• T and S have the same number of leaves, T has subtrees Ti and T 2 , Ti >T2, S has subtrees Si and 
< 52 , Si > 52 , and Ti > 5i or Ti = 5i and T 2 > S 2 

We assume that every tree T in Bn, n > 2, is presented so that Ti > T 2 , where Ti is the left subtree (or 
upper subtree if the tree is drawn with the root on the left or on the right) and T 2 is the right (or lower) 
subtree. 

For each tree T G Bn, we can identify its automorphism group A{T) as follows. Fix a labeling on the 
leaves of T using the numbers 1, 2,..., n. Label each internal vertex by the union of the labels for each of 
its children. The edges in T are pairs of subsets from [n] := {1,..., n}, each representing the label of a child 
and its parent. Let v = [u(l),w(2),... ,v{n)] be a permutation in the symmetric group ©„. Then, v G A(T) 
if permuting the leaf labels by the function i 1 —>■ v{i) for each i leaves the set of edges fixed. 

A theorem due to Jordan m tells us that if T is a tree with subtrees Ti and T2, then A{T) is isomorphic 
to A{Ti) X A(T2) if Ti y^T2, and to the wreath product A{Ti)lZ2 if Ti =T2. Since the automorphism group 
of a tree on one vertex is trivial, this implies that the general A(T) can be obtained from copies of I2 by 
direct and wreath products (see [16j for more details). Furthermore, if Ti ^ T 2 , then the conjugacy type of 
an element of A{T) is U A^, where A* is the conjugacy type of an element of A{Ti), i = 1 , 2 , and A^ U A^ 
is the multiset union of the two sequences written in decreasing order. If Ti = T 2 , then for an arbitrary 
element of A{T) either the leaves in each subtree remain in that subtree, or all leaves are mapped to the 
other subtree. The conjugacy type of an element of A(T) is then either A^ U A^, where A* is the conjugacy 
type of an element of A{Ti), i = 1 , 2 , or it is 2A^, where A^ is the conjugacy type of an element of A{Ti). In 
particular, the conjugacy type of any element of the automorphism group of a binary tree must be a binary 
partition. 

Next, we define tanglegrams. Given a permutation v G ©„ along with two trees T, 5 G Bn each with 
leaves labeled l,...,n, we construct an ordered binary rooted tanglegram {T,v,S) of size n with T as the 
left tree, 5 as the right tree, by identifying leaf i in T with leaf v{i) in 5. Note, {T,v,S) and [T',v',S') are 
considered to represent the same tanglegram provided T = T', S = S' as trees and v' = uvw where u G A{T) 
and w G A(5). Let T„ be the set of all ordered binary rooted tanglegrams of size n, and let tn be the number 
of elements in the set Tn. For example, t^ = 2 and ^4 = 13. Figures [Hand [2] show the tanglegrams of sizes 
3 and 4 where we draw the leaves of the left and right tree on separate vertical lines and show the matching 
using dashed lines. The dashed lines are not technically part of the graph, but this visualization allows us to 
give a planar drawing of the two trees. 

We remark that the planar binary trees with n > 2 leaves are a different family of objects from Bn that also 
come up in this paper. These are trees embedded in the plane so the left child of a vertex is distinguishable 
from the right child. The planar binary trees with n + 1 leaves are well known to be counted by Catalan 
numbers 

1 / 2 n\ _ 2 "( 2 n- 1 )!! 

n + 1 \ n / (n + 1 )! 

because they clearly satisfy the Catalan recurrence 

Cn — eoCn — l “t“ CiC ^_2 “t“ e2Cn—3 T * * * “t“ Cyi — iCg 

with Co = Cl = 1. For example, there are C 2 = 2 distinct planar binary trees with 3 leaves which are mirror 
images of each other while 63 = 1. The sequence of c„’s for n > 0 begins 


1,1, 2, 5,14,42,132,429,1430,4862, 
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see [HI A000108]. 

Dulucq and Guibert [7] have studied “twin binary trees”, which are pairs of planar binary trees with 
matched vertices. This is the planar version of tanglegrams. They show that twin binary trees are in 
bijection with Baxter permutations. The Baxter permutations in 5'„ are enumerated by a formula due to 
Chung-Graham-Hoggart-Kleiman [3] 

_ l^k=l\k-l)\ k )\k+l) 

rr)(T) 

See also the bijective proof by Viennot [53], and further refinements [SllEj. 


3. Proof of the main theorems 


The focus of this section is the proof of Theorem [1] namely that 

^ (2(A H-l-Af(A))-l) 

kn — 2 j 1 

X 

where the sum is over binary partitions of n. The proof of Theorem [1] reflects the chronological steps of 
discovery. Theorem will follow from a auxiliary result, and the proof of Theorem |3| is similar and is 
included at the end of the section. 

The number of tanglegrams is, by definition, equal to 


T S 


where the sums on the right are over inequivalent binary trees with n leaves, and C{T, S) is the set of double 
cosets of the symmetric group G„ with respect to the double action of A(T) on the left and A(S') on the 
right. Let us fix T S Bn and S € Bn and write C = C(T, S). Then 


ici = E1 = E 

cgc cec 


M 

|c| 


EE 

C&CweC 


1 


E 

wG&n 


1 

1^’ 


where Cw is the double coset of G„ that contains w. It is known (e.g. m Theorem 2.5.1 on page 45 and 
Exercise 40 on page 49]) that the size of the double coset Cw = A{T)wA{S) is the quotient 

\A{T)\ ■ |A(^)| 

\A{T)nwA{S)w-^\’ 

and therefore, 

^ |A(r) nwA(5')w-^| 

' ' |A(r)|.|A(5)| ■ 

We have 

|A(r) nu;A(5')w"^| = E E E lu = wvw-^^ Y E E 

wG6„ u)GS„ ueA(T) uGA(S) u£A{T) v£A{S) w€6„ 


where !•] is the indicator function. Now u = wvw~^ can only be true if u and v are permutations of the 
same conjugacy type A, which must necessarily be a binary partition as noted above. Furthermore, if u and 
V are both of type A, then there are zx permutations w for which u = wvw~^. That means that 


( 2 ) 


\C{T,S)\ 


EjAl(r);,|-|A(^)A|-Z;, 

|A(r)|.|A(A)| 


where A(T)x (respectively, y4(S')A) denotes the elements of A{T) (resp., A{S)) of type A. 

Equation ([2]) is already quite useful for computing all tanglegrams with fixed left and right trees. For 
example, if T and S are both the least symmetric tree with only one cherry, then A{T) = A(S') = {id, (1, 2)}, 
the sum is over only two binary partitions of size n, namely and (2,1,..., 1), and we get 

1^1 n! + 2(n-2)! (n^ - n + 2)(n - 2)! 

~ 2-2 ~ 4 


In some other cases the summation is over many more A’s, and can get quite complicated. 
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However, to get the formula for we want to sum Equation m over all pairs of trees, and fortunately a 
change of the order of summation helps. Indeed, we have 

/QN , _ Sa I^(’^)aI ■ I^(‘5')a| 

””t 5 |A(r)|.|A(5)| \A{T)\-\A{S)\ 



and the main theorem will be proved once we have shown the following proposition. 


Proposition 4. For a binary partition A, 

\a{tu 


E 

T^Br, 


Ili=2 (2 (Ai H-h A^(a)) - 1) 


\A{T)\ 


zx 


where A{T)x denotes the elements of AlT) of type A. 


The proposition also implies Theorem [21 as 

Y^, _ Y^ Y^ I^(^)aI _ Y^ Y^ I^(^)aI 

z^ -Z^Z^ \A{T)\ " V T ■ 


If A = 1", then |H(T)a| = I for all T G Bn, so the proposition is saying that 




(2n - 3)!! _ Cn-i 

n\ “ 2"-i■ 


This is equivalent to /\A{T)\ = c„_i. Since 2'^~^/\A{T)\ counts all planar binary trees isomorphic 

to T, this is just the well-known fact that there are c„_i planar binary trees with n leaves. 

For a general A, however, the proposition is far from obvious. What we need is a recursion satisfied by the 
expression on the right, analogous to the recursion c„ = cgCn-i + ciCn-i -b • • • -b c„_iCo for Catalan numbers. 


Lemma 5. For a nonempty subset S = {ii < 12 < ■ ■ ■ < ik} of the natural numbers define 

(5) rs(a;i,a;2, ■ • ■) = (xi^ H-f - I)(a;i3 ^- ^ ^ik - ’ {xt^-i + a;^ - ^)ix^,^ - !)• 

Let n>2, let x denote variables Xi,X 2 , ■ ■ ■, and let x/2 denote xi/2, X 2 / 2 ,.... Then 

r[„](x) = 2"“V[„](x/2)-b ^ rs(x) •r[„]\s(x). 

lGSC[n] 


Example. For n = 3, the lemma says that 


{x 2 + X 3 - l)(a;3 - 1) = {x 2 -b a;3 - 2)(a;3 - 2) -b 1 • ( 0:3 - 1) -b (a;2 - 1) • 1 + (a;3 - 1) • 1, 


where the last three terms on the right-hand side correspond to subsets {!}, {1,2}, and {1,3}, respectively. 
As another example, take Xi = 2 for all i. Then rs{x.) = (2|S'| — 3)!! (where we interpret (—1)!! as 1), 
rsf^/2.) = 0, and by the obvious symmetry of S and [n] \ S the lemma yields 


2- (2n 



(2fc-3)!!(2n-2A;-3)!!, 


which is equivalent to the standard recurrence for Catalan numbers. 


Proof of Lemma\^ The proof is by induction on n. For n = 2, the statement is simply 312 — 1 = {x 2 — 2) -bl • 1. 
Assume that the statement holds for n — 1, and let us prove it for n. Both sides are linear functions in X 2 , so 
it is sufficient to prove that they have the same coefficient at X 2 and that they give the same result for one 
value of a; 2 . 

The coefficient of X 2 in r[„](x) (resp., 2"“^r[„](x/2)) is clearly r[ 2 ^„](x) (resp., 2"“^r[2_„](x/2)). On 
the other hand, rsfx.) ■ r[ji]\ 5 (x) contains X 2 if and only if 2 £ 5, in which case the coefficient at X 2 is 
^S\{i}(x) ■ C[ 2 _„]\s(x). The coefficients on both sides are equal by induction. 
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Plug the value X 2 = 2 — xs — ■ ■ ■ — Xn into both sides. Clearly, the left-hand side becomes It 

is easy to see that if 2 e S', then rs{^) ■ r[„]\ 5 (x) -|- r 5 \{ 2 }(x) • T([„]\ 5 )u{ 2 }(x) = 0. That means that all the 
terms in the summation cancel out except r[„]\{ 2 }(x) ’ ^{ 2 }(x) = ^[ri]\{ 2 }(x). Obviously, r[„](x/2) = 0, so the 
right-hand side also equals r[„]\{ 2 }(x)- D 

Proof of Proposition^ Say A is a binary partition of n. The proof is by induction on n. For n = 1, the 
statement is obvious. Assume that the statement holds for all binary partitions up to size n — 1. Our task 
is to show 

|A(r)A| _ 7'[f(A)](2Ai, 2 A 2 , 2 A 3 ,...) 

by showing the left hand side satisfies a recurrence similar to ©• 

Given T G Bn, let Ti and T2 be the subtrees of the root in T. Fix a labeling on the leaves of T such 
that the leaves of Ti are labeled [1,/c] and the leaves of T 2 are labeled [k -I- l,n]. Consider each A{Ti) to 
be a subgroup of the permutations of the leaf labels for Ti. We can obtain a permutation of type A in 
A{T) in one of two ways. First, we can choose permutations wi G A{Ti),W2 G A{T2) of types A^ and A^, 
then W1W2 is a permutation of A(T) of type A. Second, if all parts of A are at least 2 and Ti = T2 (and in 
particular n = 2k), we can choose an arbitrary permutation wi G A(Ti) and another permutation W 2 G A[Ti) 
specifically of type A/2 := (Ai/2, A 2 / 2 ,.. .) and construct a permutation w G A{T) of cycle type A as follows. 
Say / : [1,/c] —> [k + l,n] mapping i to i k induces an isomorphism of Ti and T 2 . Define the “tree flip 
permutation” tt to be the product of the transpositions interchanging i with f(i) for all 1 < * < /c. Now take 
the product 

W = TTlCl 717(12 ■ 

It is clear that w G A{T) since it is the product of permutations in A{T). Observe also that the cycles of 
w are constructed so the leaf labels of Ti interleave the leaf labels of T 2 in the cycles of W 2 so w will have 
cycle type A. For example, if A = (6,4), then |A| = 10 and tt = (1 6)(2 7)(3 8)(4 9)(5 10). If we choose 
wi = (1 4)(2 5)(3) and W 2 = {6 9 7)(8 10) then w = TTWiTTwf^TTW 2 = (6 1 9 5 7 4)(8 2 10 3), all in cycle 
notation. Also, every element of A{T) is constructed in one of these two ways. 

We need to be careful to differentiate between the cases when the subtrees Ti,T 2 are different and when 
they are equivalent. We have 


I^('^)a| _ I^('1")a| y^ I^('1 ")a| 

V \MT)\ |A(r)| |A(r)| 


y^ / y^ I"^('^i)ai| • |^('12)aH \ 
t^tAa7u%^a \^( T ,)\.\ AiT2 )\ ) 


y^ (Z]aiua^=a |A(ri)Ai| • |A(Ti)a 2|) + |A(Ti)|. |A(ri)A/2| 

^ 2|A(Ti)P 


or equivalently 


( 6 ) 


2 y^ I"^(^)a| 


y^ I^('1 "i)a/2| 


E 

AiuA2=A 



\A{T,)\ j 



|A1(T2)a2| \ 

I^(7’2)| j ■ 


Let 

nii^2^( 2 (Ai -I- ■ • ■ -I- A^(a)) — 1 ) r[^(A)]( 2 Ai, 2A2, 2A3,...) 

Q\ = -=-; 

2a 2a 

the notation also makes sense if A^(a) = 1/2, as in that case q\ = 0. By the induction hypothesis and ([6]), it 
suffices to prove that 


29 a — 9 a /2 + E 

A1uA2=A 


(7) 


9ai ■ 9a2■ 
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After multiplying both sides by z\, this is 


^(A) 


£(A) 


2 J^(2(Ai + • • • + A^(;^)) — 1) — 2^*-'’'^ + • • • + ^£(\) — 1) 



+ f > 2 ) ■ II -^ ^l(Ai)) “ 1 ) ■ H (^(^i ^ -^^(A^)) - 1 )> 




where (^i\2) = Hi equality holds by Lemma [S] with Xi = 2 Ai. 


□ 


We conclude this section with the proof of Theorem |31 

Proof of Theorem\^ Let T = (Ti, T 2 ,..., T^) be an ordered list of binary trees in Bn- Define C'^ to 
be the set of “multicosets” of 6„ with respect to A(Ti) x A{T 2 ) x ••• x A{Tk). More concretely, given 
{wi,... ,Wk-i),{w[,... e we say {wi,... ,Wk-i) =t (w'-y,... provided there exist 

ti G A{Ti) such that Wi = for alH = 1,..., /c — 1. Then, is the set of equivalence classes modulo 

=T- By definition, the number of tangled chains of length k and size n, denoted t{k,n), is given by 



( 8 ) 


where the sum is over all ordered lists T = (Ti, T2,..., T^) of trees Ti € Bn- 

Fix a particular list of trees T = (Ti,T2,...,T^), and let C'^{wi, -. - ,Wk-i) be the multicoset in 
containing (wi,..., Wk-i)- Clearly, 



We give a recurrence for |C’^(wi,... ,'u;fe_i)| in terms of the following subgroup. Let A{C'^{wi ,... ,Wk-i)) 
be the subgroup of all ti G A(ri) such that there exist ti G A{Ti) for 2 < f < fc satisfying Wi = tiwAi^i for all 
i = 1 ,..., k— 1 . In this case, {tiWi,W2, - - -, Wk-i) =t {wi,W2, ■ ■ •, Wk-i) so we think of A^C'^{wi ^..., Wk-i)) 
as the “left automorphism group” of C'^{wi ,... ,u;fc-i). Observe that 


A{c'^{wi ,..., rcfe-i)) = A(ri) n wiA{T2)w^ ^ fl • • • fl W1W2 ■ ■ ■ Wk-iA{Tk)Wf.^^ • • ■ ^2 ^'“'1 ^ 


|A(C'^(?iIl, . . . ,Wk-l))\ = ^ X! “ WlOWi • [O = W2t3W2 • • • Itk-l = J. 


i=l U&A(Ti) 


Now let T' = (r2, • ■ ■, Tk)- For each (u2,..., Vk-i) G (u>2, ■ ■ ■, Wk-i)^ we can prepend a vi to create a 
distinct element (ui, t>2,. •., Vk-i) G C'^{wi ,..., Wk-i) exactly when vi is in A{Ti)wiA{C"^ {w2, - - -, Wk-i)) 
which is again a double coset of 6 „. Thus, by the formula for double cosets we have 



by induction on k- Therefore, 


( 9 ) 



where the denominators do not depend on the wfs- 
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Focusing on the sum in the numerator in ([S]), we have 

{wi,W2,...,Wk_i) 

= X! X! ■ ■ ■ X! 1^1 = Wit2W^^j ■ ■ ■ ftk-l = Wk-ltkW^l^l 

{wi,W2,...,Wk_i) tieA(Ti) tk&A{Tk) 

= X! ■■■ X! X! Ih = ■ ■ ■ ftk-l = Wk-ltkW^l-Ji 

tieA(Ti) tkGA(Tk) (wi,u} 2 ,...,Wk-i) 

and so with similar logic as before, noting that the summand will be nonzero exactly when ti, t 2 , ■ ■ ■ ,tk are 
all of the same conjugacy type A, 

j:,\AiT,),\-\A{T2)x\---\A{Tkh\-z^,-^ 


(10) IC^I = 

Plugging m into dH), we obtain 


\A{T,)\-\A{T2)\---\A{Tk)\ 


t{k,n)= 

= E^r-fE 


j:^\A{T,)^\-\AiT2)x\---\AiTkh\-z^ 


k-1 


IA(T,)l-IA(T2)l---IA(n)l 


\TeBn 


\A{T)\ 


and Theorem [3] now follows from Proposition 01 


□ 


4. Random generation of tanglegrams and inequivalent binary trees 

In this section, we describe an algorithm in 3 stages to produce a random tanglegram in r„. The stages 
are based on Equation (j3|) and the proof of Proposition HI A similar algorithm is also described to choose a 
random binary tree with n leaves. In this section, “random” will mean uniformly at random unless specified 
otherwise. 

Recall from Section [3] that if T is a tree with equivalent left and right subtrees, we denote by tt the “tree 
flip permutation” between the subtrees. Also, for a partition A, we defined 

nii^2^ + • • • + A£(a)) - 1) 

q\ = -. 

The q\ notation also makes sense if Xi(\) = 1/2, as in that case q\ = 0. 

Algorithm 1 (Random generation of w G A{T)). 

Input: Binary tree T G R„. 

Procedure: If T is the tree with one vertex, let w be the unique element of A{T). Otherwise, the root of T 
has subtrees Ti and T 2 . Assume the leaves of Ti are labeled [1, k] and the leaves of T 2 are labeled [fc + 1, n]. 
Use the algorithm recursively to produce Wi G A{Ti), i = 1, 2 where A(Ti) is a subset of the permutations of 
[l,n] which fix [A: + l,n] and A(T 2 ) is a subset of the permutations of [l,n] which fix [l,fc]. Construct w as 
follows. 

• If Ti ^ T 2 , set w = wiW 2 - 


• If Ti = r 2 , choose either w = W 1 W 2 or w = t:wiW 2 with equal probability. 
Output: Permutation w G A{T). 


Algorithm 2 (Random generation of T with non-empty A{T)\ and w G A{T)\). 

Input: Binary partition A of n. 

Procedure: If n = 1, let T be the tree with one vertex, and let w be the unique element of A{T). 
Otherwise, pick a subdivision (A^, A^) from {(A^, A^): A^ U A^ = A} U {(A/2, A/2)}, where (A^, A^) is chosen 
with probability proportional to q\iq \2 and (A/2, A/2) with probability proportional to q\/ 2 - 
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• If A^, A/2, use the algorithm recursively to produce trees Ti,T 2 and permutations wi G A{Ti)x^, 

W 2 G A{T 2 )\ 2 . If necessary, switch Ti ga T 2 , wi ga W 2 so that Ti > T 2 . Let T = (Ti,T 2 ), w = wiW 2 - 

• If A^ = A^ = A/2, use the algorithm recursively to produce a tree Ti and a permutation W 2 G A{Ti)x/ 2 j 
and use Algorithm [T] to produce a permutation wi G A(Ti). Let T = (Ti, Ti) and w = •kwittWi^ttw 2 - 

Output: Binary tree T and permutation w G A{T)\. 


Algorithm 3 (Random generation of tanglegrams). 

Input: Integer n. 

Procedure: Pick a random binary partition A of n with probability proportional to zxq\ where tn = 

Use Algorithm [5] twice to produce random trees T and S and permutations u G A(T)x, v G A(5')a- Among 
the permutations w for which u = wvw~^, pick one at random from the zx possibilities. 

Output: Binary trees T and S and double coset A{T)wA{S), or equivalently (T, re, S). 


Algorithm 4 (Random generation of T G Bn). 

Input: Integer n. 

Procedure: Pick a random binary partition A of n with probability proportional to qx- Use Algorithm [5] to 
produce a random tree T (and a permutation u G A(T)x)- 
Output: Binary tree T. 

Algorithm S] is not the first of its kind, see also [5] . 

Algorithm 5 (Random generation of tangled chains). 

Input: Positive integers k and n. 

Procedure: Pick a random binary partition A of n with probability proportional to z^~^q^ where t(k,n) = 
^x~^*lx- Use Algorithm[2]A: times to produce random trees p and permutations Ui G A(Ti)x for i = 1,... ,k. 
Among the permutations Wi for which Ui = pick one uniformly at random for each i = 1,..., k—1. 

Output: (Ti, ...,Tk) and (ici,... ,Wk-i)- 


Theorem 6. For any positive integer n, the following hold. 

• Algorithm\^ produces every permutation w G A{T) with probability 

• Algorithm\^ produces every pair {T,w), where w G A{T)x, with probability 

• Algorithm[^ produces every tanglegram with probability -A. 


• Algorithm^ produces every inequivalent binary tree with probability 

• Algor ithm\^ produces every tangled chain of length k of trees in Bn with probability 

Proof. The first two proofs are by induction, with the case n = 1 being obvious. The induction for Algorithm 
[T]is trivial. 

For Algorithmic! say that we are given a binary partition A, a tree T with n = |A| leaves, and w G A(T)a. 
We compute the probability that Algorithm [2] produces T and w. Assume first that Ti > T 2 are the subtrees 
of T. In particular, that means that w can be written uniquely as W 1 W 2 , where wi G A(ri) and W 2 G A(T 2 ). 
Say that Wi is of type A®; we must have A = A^ U A^. If A^ ^ A^, there are two ways in which Algorithm [2] can 
produce (T, w): either we partition A into (A^, A^), and then the algorithm produces (Ti, wi) and {T 2 , W 2 ), or 
we partition A into (A^, A^), then the algorithm produces (T 2 , UI2) and (Ti, wi), and finally switches Ti gg T 2 , 
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wi W 2 - Since Ti and T 2 are chosen independently, we can apply © and induction to obtain the probability 
that (T, w) is chosen, namely 

1 1 _ 1 _ 1 

2qx ■ \A{T,)\ • q^^ ■ \AiT 2 )\ • q^. ~ |^(Ti)| ■ \A{T 2 )\ • qx ~ \A{T )\• qx ’ 

If = A^, but Ti 7 ^ T 2 , there are again two ways in which Algorithm [2] can produce {T,w): we must 
partition A into (A^, A^), and then it can either produce (Ti,wi) and {T 2 ,W 2 ) or {T 2 ,W 2 ) and (Ti,t(;i); in the 
latter case it switches Ti T 2 , Wi t-A W 2 - Similarly, the probability is 

Now assume that Ti = T 2 . Either w can be written as W 1 W 2 , where wi € A(Ti)xi and W 2 € A(T 2 )> 2 , or 
as 7 TW 27 rw 2 ^ 7 TWi, where wi € A(Ti)x /2 and W 2 € A(Ti). In the first case, (T,w) is produced with probability 

1 1 _1_ 1 

2qx ■ \AiT,)\-qxi ■ \AiT,)\-qx 2 " 2 • |A(Ti)|2 - 9 ^ " \AiT)\-qx 
In the second case, it is produced with probability 

qx /2 1 1 _ 1 _ 1 

2qx ■ |A(ri)|.q ;,/2 ■ |A(Ti)| ~ 2 ■ \A{Ti)\^ ■ qx ~ \A{T)\ ■ qx 
This finishes the case for Algorithm [21 

The proof of the statement for Algorithm [3] is essentially just a rewriting of the proof from Section |3l we 
include it for completeness. We are given n and a tanglegram {T,w,S) with T and S binary trees with n 
leaves, C = A{T)wA{S) the double coset containing w with respect to A(T) and A(S'), and we want to prove 
that P{T, S, C), the probability that this triple is produced by Algorithmic is l/tn- 

We proved that ^ zxq\ = tm so the probability of choosing a binary partition A is zxq^/tn- So we have 

P(T,5,C)=^^P(T,5,C|A), 

A 

where P{T, S,C\X) is the conditional probability that {T,S,C) is produced if A is chosen. We can further 
condition the probability: P(T, S,C\X) = ^ P(T, S', Clu, v, T, S', A) • P{u,v,T, S\X), where the sum is over 
u S A{T)x, V G A{S)x- Furthermore, 


and so 


P(T, S, C\u, V, T, S, A) = P{C\u, v) and 


P(u, V, T, S|A) = P(T, m|A) • P(S, u|A), 


P{T,S,C) 


A " uGA{T)xvGAiS)x 


1 

\A{T)\-qx 


1 

l^('S')| • qx 


1 

T:-2^\a{t)\-\a{s)\ 


E E 

u(^A{T)xvGAiS)x 


|C'nP“'’'| 


where P“>’' = {w G ©„: u = wvw ^}. We know that IP”’”] = zx, so 


p(r,s,c) = --^ 


E E = 


^ |A(r)| • |A(S)| 

A ' ^ ^ ueA{T)xv&A(S)xweC 

1 




" weC A ' 

_ 1 IA(r)A n wA(s)a'u: ^ 

wee A 

= e-E 


u = wvw 




1 


^n. ^ Cit 

w€C 


\A{T)\-\A{S)\ 

1 

in. 






Finally, let us prove the statement for Algorithmic We have 

|a1(T)a| Ea^IPM-I 

■ bj 


p(r) = 5]p(r|A)-p(A) = ^ 


\A{T)\-qx b„ 


\A{T)\ 


A A 

which proves that Algorithmic produces every inequivalent binary tree with the same probability. The proof 
for Algorithm [5] is similar to Algorithms |C and |C so we omit the formal proof. □ 
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5. Asymptotic expansion of 


In this section, we use Theorem [T] to obtain another formula for and several formulas to approximate 
tn for large n. 


Corollary 7. We have 


( 11 ) 


“ 4n-l ■ 


n{n - 1) • • • - ImI + 1) 


M • nf=i - 2(^1 + • • • + - 2j - 1)2 ’ 

where the sum is over binary partitions yt with all parts equal to a positive power of 2 and |/J.| < n including 
the empty partition in which case the summand is 1. 

Proof. Every binary partition A of size n can be expressed as where all parts of p, are at least 2. We 

have z\ = z^{n — |^|)! and 

t{X) ^(A)-l 

^ (2(Ai + • • • + — l) = (2(n — Ai — • • • — Ai) — 1) 

i=2 i=l 

= (2(n-/ri-/li) - 1) • (2n - 2|/i| - 1)!! 

i=l 

(2n-3)!! 

- 2(^1 +...+^.-i) - 23 - 1) ■ 

Since (2n — 3)!!/n! = Cn-il2,'^~'^, (fTTI) is an equivalent way to express the number of tanglegrams. 

The first few terms of the sum corresponding to partitions 0, (2), (4), (2,2), (4,2), (2,2,2), (8) are 
n(n — 1) n(n — l)(n — 2)(n — 3) n(n — l)(n — 2)(n — 3) 

2(2n- 3)2 4(2n- 3)2(2n- 5)2(2n- 7)2 8(2n - 3)2(2n - 7)2 

n(n — l)(n — 2)(n — 3)(n — 4)(n — 5) ^ n(n — l)(n — 2)(n — 3)(n — 4)(n — 5) 


8(2n-3)2(2n-5)2(2n-7)2(2n-ll)2 48(2n - 3)2(2n - 7)2(2n - 11)2 

n{n — l)(n — 2)(n — 3)(n — 4)(n — 5)(n — 6)(n — 7) 

8(2n - 3)2(2n - 5)2(2n - 7)2(2n - 9)2(2n - ll)2(2n - 13)2(2n - 15)2 ' 

Corollary 8. We have 


tn 


nl 


esc: 


n —1 


634” 


An — l 


and 


tn 


TTU'^ 


V^-e^- 

We can also compute approximations of higher degree. For example, we have 


tn — 


— 2 I 

qri-l 

22 "-i . n" 

0F-e"- 


1 + 


1 137 1285 456017 6140329 , .gx 

4^ 256 n2 + 1024 131072 524288 ^ 

931423 826301423 


211060350013 

12 n ' 2304 n2 ' 414720 ' 159252480 ' 13377208320 n® 


13 3089 

1 + + 


□ 


+ 0(n-) . 


Sketch of proof. The crucial observation is that 
n{n - 1) • • • (?T- - ImI + 1) 

- 2(mi + • • • + M.-i) - U - 1)2 


^\u\ 

Zf, ■ (2n)2(fol-dfo) 


1 

2^(fol—^( a)) . 


So, to find an asymptotic approximation of order 0{n ^”^) or 0(n ^), we only have to consider 

partitions p, with \fj.\ — 2£(fi) < 2m in Equation pil) . For m = 0, we only consider partitions of the type 
22 • • • 2. The contribution of /r = 2^ is l/(22''2^fc!), and the sum converges to = e®. 

Similarly, the coefficient of n~^ can be obtained by considering the coefficient of n~^ in each of these 
terms, and the higher terms by considering in turn partitions of type 42^, 4^2^, 4^2^, 82^, etc. The last 
expansion is obtained by considering the asymptotic expansions of c„_i and n!. □ 
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6. A RECURRENCE FOR ENUMERATING TANGLEGRAMS AND TANGLED CHAINS 


In this section, we give a recurrence for computing Recall that for each nonempty binary partition 
A, we can construct its multiplicity vector = {mo, mi, m 2 , rn^,.. .) where is the number of times 2* 
occurs in A. The map A i-A m^ is a bijection from binary partitions to vectors of nonnegative integers with 
only finitely many nonzero entries. The quantity zx for a binary partition A is easily expressed in terms of 
the multiplicities in m^ as 

rrih 

2A= rnhi= n 

h>0 h>0 j—1 

rrih^O 


We will use the functions 


( 12 ) 


f{s) := (2s-l)^ 


(13) 

and 

(14) 


with base cases 

(15) 


c{h,m,s) := 

i=i 


P{s+j ■ 2p 

j. 2 ^ ’ 


r{h, n, s) 


n 


^ c{h,m,s) 

m—0 

(n—m) even 


h + 1, ■ 


, s + m2 


c{h, 0, s) = r{h, 0, s) = 1. 


Lemma 9. For n > 1, the number of tanglegrams is 

^ _ r(0,n,0) 

” Pin) ’ 

which can be computed recursively using m- 
Proof. Let := (1 — 2n)‘^tn. By the main formula 

(16) P = ^ nfi^i (2(A. + --- + A,(a))-1)' ^ 

A 

where the sum is over binary partitions of n. 

We will consider the contribution to (1161) from the parts of the partition of size 2^ for each h separately. 
To do this we will need to keep track of the partial sums of parts smaller than 2^. Let s^ = (sq , ,...) where 

mi2* and Sg = 0. Then the contribution of the parts of size 2^ in A to the corresponding term in 
(HU) is the factor c{h,mh,sp. Using this notation, we have 

(17) in= ^ c(0,mo,0)c(l,mi,Si)c(2,m2,S2)--- 

m^ — {mQ ,mi ,...)l-n 

where the sum is over binary partitions of n represented by their multiplicity vector. 

Next consider the binary partitions with exactly j parts of size 1. Note n — j must be even for this set 
to be nonempty. The binary partitions of n with exactly j parts equal to 1 are in bijection with the binary 
partitions of so 


(18) 


— 


E 

rriQ—O 

(n—mo) even 


c(0,mo,0) 


E^ 

(mi,m2, 


c(l,mi,mo)c(2,m2,mo + 2 • mi) • 
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Observe that the recurrence in (|14l) gives rise to the expansion 

r{h, n,s) = ^ c{h, nih, s)c{h + 1, mu+i, s + mu ■ 2^)c{h + 2, mh+ 2 , s + nih ■ 2^ + ruh+i ■ 2'“+^) • • • 

where the sum is over binary partitions of n but the indexing is shifted so ruh is the number of parts of size 
1. Thus, 

in= ^ c(0,m,0) r(l, ^ = r(0,n,0) 

m^O ^ ^ 

(n— m) even 

which completes the proof since /^(n) = (2n — 1)^. □ 


We can extend the functions above to count tangled chains: 
(19) f\s):={2s-l)\ 


( 20 ) 

and 

( 21 ) 


(h,m,s) := [[ --, 

1=1 


r^(h,n,s)-.= ^ c^(fe, m, s) r + 1, ^ ^ s + m2^ 

7Y>—n ' ^ 


m=0 
(n—m) even 


with base cases 


(22) c'=(/i,0,s) =r'=(h,0,s) = 1. 

Then a proof very similar to the case k = 2 also proves the following statement. 

Corollary 10. For n > 1, the number of tangled chains of length k is 

r^(0, n, 0) 

/'=(n) 

which can be computed recursively using m- 


7. Final remarks 

Generating functions. It is known (and easy to prove) that the ordinary generating function for inequiv¬ 
alent trees satisfies the functional equation 

B{x) = x+^ {Bixf' + B{x'^)) . 

This is, of course, equivalent to a recurrence for the sequence Given that in this paper we prove both 
explicit formulas and recurrences for the numbers of tanglegrams and tangled chains, it makes sense to ask 
the following. 

Question 1. Does there exist a closed form or a functional equation for the generating function of tanglegrams 
or tangled chains? 
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Number of cherries and other subtrees. Cherries play an important role in the literature on tanglegrams. 
For example, Charleston’s analysis [SJ pp. 325-326] suggests the following question. 

Question 2. What is the expected number of matched cherries in a random tanglegram? 

Computer experiments with random tanglegram generation suggest that the following is true. 

Conjecture 1. The expected number of cherries in the left tree in a random tanglegram converges to n/4. 

Conjecture 2. The expected number of copies of the tree T in the left tree of a random tanglegram of size n 
is asymptotically equal to where I is the number of leaves ofT and k is the number of symmetries 

of T, i. e. vertices with identical subtrees. 

It also seems that the number of copies of a tree converges to a normal distribution. 


If the conjectures hold, then for every tree T with I leaves and k symmetries, the number of copies of 
the tree with T as left and as right subtree in the left tree of a randomly chosen tanglegram asymptotically 
equals So that would imply the following. 

Conjecture 3. Let T' G Bn be the left tree of a tanglegram chosen uniformly at random. The expected 
number of generators of AlT') is asymptotically equal to 

( X! 4Z(T)+fc(r) 

\TGBn 

It is not hard to see that the sum in the conjecture equals f(^)n, where f(x) is the function defined by 
/(O) = 0 and f(x) = x + 5 /( 2 ;)^ + (a; — ^)/(a:^), or explicitly 



f{x) = 1 - 


\ 


1 - 2a; + (I - 2a;) 1 


1 - 2a;2 + 


(1 - 2a;2) (^1 - Vl - 2a;4 



Note that the computation of f{j) = 0.27I04I69360883278703... converges very rapidly: the number of 
correct digits roughly doubles after each step. 


Connection with symmetric functions. The main theorems suggest that symmetric functions might be 
at play; note, for example, the similarity with the formula hn = where hn is the homogeneous 

symmetric function, px the power sum symmetric function, and the sum is over all partitions of n. 

Question 3. Is there a connection between tanglegrams (or more generally tangled chains) and symmetric 
functions ? 

Remark. Based on a manuscript version of this paper, Ira Gessel pointed out that there is indeed a con¬ 
nection between symmetric functions and the enumeration of the ordered and unordered tanglegrams based 
on the theory of species. His claims will be spelled out in a forthcoming paper nni. 

Variants on tanglegrams. Tanglegrams as described here fit in a set of more general setting of pairs of 
graphs with a bijection between certain subsets of the vertices (more completely described and motivated in 
[16]). One can also consider unordered tanglegrams by identifying (T,v,S) with {S,v~^,T). For example, 
the 4th and 5th tanglegrams in Figure dj are equivalent as unordered tanglegrams, and so are the 8 th and 
10th. From this picture, the reader can verify that there are 10 unordered tanglegrams of size 4. 

Because of reversibility assumptions for the continuous time Markov mutation models commonly used 
to reconstruct phylogenetic trees, unrooted trees are the most common output of phylogenetic inference 
algorithms. Thus another variant of tanglegrams involves using unrooted trees in place of rooted ones. The 
motivation for studying these variants comes from noting that many problems in computational phylogenetics 
such as distance calculation between trees [ 1 ] “factor” through a problem on tanglegrams. 

Question 4. Is there a nice formula for the number of 

• unordered binary rooted tanglegrams, 

• ordered binary unrooted tanglegrams, or 

• unordered binary unrooted tanglegrams? 

These counts have been found up to 9 leaves (Table [Ij by direct enumeration of double cosets [TB] . 
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Table 1. The number of tanglegrams of various types up to 9 leaves, 
leaves rooted ord. rooted unord. unrooted ord. unrooted unord. 


1 

1 

1 

1 

1 

2 

1 

1 

1 

1 

3 

2 

2 

1 

1 

4 

13 

10 

2 

2 

5 

114 

69 

4 

4 

6 

1509 

807 

31 

22 

7 

25595 

13048 

243 

145 

8 

535753 

269221 

3532 

1875 

9 

13305590 

6660455 

62810 

31929 
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